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HOW TO IMPROVE INSTRUCTION ON THE BASIS 



OF EVALUATION? 
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The role of evaluation in improving instruction 
varies with the instructional practices of the schools as well 
as with the evaluation instruments that are available. In the 
pasty and even today in many schools ^ the material of a topic or 
unit was presented to all the pupils in the class at the same 
time and they were also given the same assignments or learning 
tasks to be completed during the time allotted to the topic. It 
was expected that some pupils would learn more and some less^ 
and as the year went by the "slow learners" would be increasing- 
ly separated from the average pupils because of the increasing 
differences in the amount they were learning. Their educational 
achievements were appraised from time to time on the basis of 
their class performance and the tests made by the teachers to 
cover the contents of the topic or unit. Published tests which 
were norm-referenced were commonly administered not more than 
once a year to provide the school administration with data re- 
garding the relative standing of the school in relation to other 
schools and to furnish teachers with information about the re- 
lative achievement of pupils in each class that could be com- 
pared with the results obtained from the tests and assignments 
given by the teachers. Consciously or unconsciously ^ the school 
operated very much like an institution for sorting children as 
well as an educational one. Inadequacies in learning in such a 
system were thought to be largely due to the limited intelli- 
gence of the pupils or their lack of effort rather than to the 
inappropriateness and ineffectiveness for some children of the 
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learning experiences employed. The so called "intelligence 
tests" or "scholastic aptitude tests" were used to identify 
pupils with different levels of intelligence and ther "IQ score^^ 
were compared with their achievement scores to identify "under- 
achievers", that is, those whose ra.iking on intelligence tests 
were higher than their ranking on achievement tests. 

In a system operating on these assumptions and 
following these procedures, certain decisions could be made 
about instruction partly based on test results. If the school's 
mean score on the achievement test was at or above the accepted 
norm, the instruction was deemed satisfactory. If it was con- 
siderably above the norm the instruction was considered superior. 
In either case, if there were a number of "underachieving 
pupils", instruction for them should be improved by increasing 
their motivation, by furnishing additional assignments, by set- 
ting up some remedial sessions, or by assigning them to a dif- 
ferent track. 

If the school's mean score was below the norm, 
the intelligence test results were examined to find out whether 
the lower performance on achievement tests could be explained 
by the lower intelligence. If the school's mean score on the 
intelligence test was as much below the norm as the achievement 
mean, the instruction was deemed satisfactory. If the intel- 
ligence test mean was much farther below the mean than the ach- 
ievement test mean the instruction was considered superior. In 
these cases, too, if thfere were a considerable number of "under^ 
achievers" the same efforts were recommended to help them "ach- 
ieve up to their capacity". If the school's mean score on the 
intelligence test was higher than the mean score on the achieve- 
ment test, it was assumed that the lower achievement was not 
due to lack of ability on the part of the average students. In 
such cases teachers and principals explored other possible ex- 
planations such as inadequate learning in the earlier grades, 
lack of effort, poor motivation, too little time allowed in the 
schedule for these subjects, inappropriate instructional mater- 
ials, and inappropriate teaching methods. Steps that were com- 
monly taken to deal with these problems were to suggest that 
teachers in the earlier grades give more emphasis to basic 
skills, to urge pupils to work harder, to try various devices 
to increase interest in these subjects, to schedule more time 
for them to establish remedial sessions or courses, to place 
students in a lower "track", to select different text books 
and work books, seeking particularly to find materials with 
lower levels of vocabulary, and to try another teaching method- 
ology. 

The major concern in these procedures was to help 
students do better whose rank on achievement tests was lower 
than their rank on intelligence or scholastic aptitude tests. 
Very little attention was given to those whose achievement rank 
was higher. Perhaps this was partly due to the difficulty of 
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rationalizing an "over achiever" concept. How could a student 
do better than he was able to do? Perhaps it was partly due to 
the fact that little outside pressure was placed on the teacher 
to help "fast learners" to achieve more. 

It is clear that norm-referenced tests have been 
useful in instructional systems designed to sort students dnd 
to push out or place in lower tracks students whose achievement 
ranks are lower. It is also true that most teachers used other 
devices as part of their procedure for aiding individual stu-- 
dents with their learning difficulties. The students perfor- 
mance in class, his handling of assignments, his workbook ef* 
forts, the marks he made on tests given by the teacher were com- 
mon indicators to the teacher of the student's learning problems. 
But without an accepted doctrine that all or almost all students 
could be expected to master what was presented the teacher had 
no approved procedure for analyzing the inadequacies of the 
student's performance. Generally, the student was told that he 
did not perform successfully on a particular learning task and 
was directed to try again. If the second or third trial was 
still unsuccessful, the busy Leacher usually gave up his atteitpt 
to give individual help to the student. Fortunately, in some 
cases where the student's specific unsatisfactory performance 
was identified and he was asked to try again, the second or 
third effort was successful so that not all students were sort- 
ed and assigned sections or tracks solely on the basis of norm 
referenced tests. 

The major point, however, should be clear, namley 
.that an educational system that views sorting as a major func- 
tion will find norm-referenced tests helpful. But schools are 
no longer accepted as adequate if they operate primarily to 
sort students. Currently schools are expected to educate all 
their pupils; yet most are bound to a system in which all learn- 
ers are expected to move at the same rate and those performing 
less satisfactorily than others are discouraged and give up or 
get farther and farther behind so that they lack the earlier 
achievement required to perform the later learning tasks. 

We kno^^ enough about the way in which complex 
behavior is learned to realize that the learner is aided by 
starting with behavior he can readily acquire, and after he 
masters the first step he moves on step by step. He masters 
each new step if it is largely based on the preceding one. 
Complex behavior is acquired through a sequence of steps. An 
adept learner works out his own sequence but many children need 
help, and for them it is necessary to devise an effective learn- 
ing sequence. 

With the current expectation that the school 
will educate all children the instructional practices and the 
supporting doctrine cire changing from those that operate in a 
sorting system. The delay in developing a new system has been 
due to the fact that the old system has worked quite well with 
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a majority of children so that it has not b^en questioned but 
rather we have talked about "slow learners" or "dull students", 
now we are recognizing the need to reconstruct the learning 
system if we are to reach effectively the children who have not 
been learning previously. In the new systems, behavioral ob- 
jectives arranged in sequence, are being formulated and learn- 
ing experiences selected or devised to enable students to attain 
these objectives in step-by-step progression. Such a 5jystem can 
be used in group instruction as well as in individual tutoring, 

A Teacher who tries to manage a learning system 
where every child is engaged in activities that he can perform 
finds three evaluation instruments of great help. Since such a 
system requires the arrangement of learning experiences in se- 
quence for each of the major objectives, and since children dif- 
fer in the extent of their previous learning, for each student, 
a decision must be made regarding a place in the sequence where 
he can begin. This decision is aided by a placement procedure 
or test that indicates what the student has already learned and 
what next step in the sequence he can expect to carry on suc- 
cessfully. After he has engaged in the learning activities de- 
signed for the step where he begins, a mastery test can furnish 
evidence that he has acquired the desired behavior and is ready 
to proceed with the next step in the sequence. If the mastery 
test indicates that he has not acquired the behavior that this 
step in the sequence is designed to help him learn, a diagnos- 
tic test for this step can be used which enables the pupil and 
teacher to identify the particular difficulty the student is 
having and furnishes a basis for another series of learning 
experiences that provide more gradual development and further 
practice of the behavior that the student did not master during 
his first experiences with this step in the sequence. 

Current widely used achievement tests do not pro- 
vide dependable information either for placement or for mastery 
purposes, and thus far, few diagnostic tests have been developed 
for the several steps in the sequences of the major school sub- 
jects. The survey types of achievement tests have been designed 
to measure individual differences among students, not to indicate 
where each one is on a learning sequence, or whether he has 
gained mastery of the step on which he has been working, and, of 
course, survey tests do not furnish a diagnosis of the specific 
learning difficulty the student is encountering. The items 
used in achievement tests in a given subject are designed to 
sample the knowledge and skills commonly emphasized in the sub- 
ject over several school grades. On the basis of try cuts, 
items are selected which are answered correctly by a greater 
percentage of children in each higher grade; for example, a 
greater percent of fifth grade children than fourth grade child- 
ren. Items are rejected that are answered correctly by most of 
the pupils or by few of them, since they do not discriminate 
among the pupils <r and thus do not serve to arrange them in order 
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from those who get high scores to those who receive low ones. 
The result of this selection procedure is a test which does 
not reliably sample each objective and each step in a sequence. 

A placement procedure useful for a teacher and 
the students in a learning system designed to help every child 
progress involves a two-step appraisal. The first is a test 
constructed to sample each major section of the learning se- 
quence. A primary reading program, for example, may have a se- 
quence in which the first several steps involve the use of 
simple oral communication, the second several steps are con- 
cerned with the alphabet, and the third, with the decoding of 
common phonemes. The first placement test would be constructed 
to sample reliably each of the three series of steps but would 
not furnish a reliable sample of the behavior in each step. It 
would provide, thereby, an indication of the major divisions in 
the sequence already learned and in what division the beginning 
step would be found. This enables the pupil to check himself 
on mastery tests within this division to identify the next step 
that he has not yet mastered. 

When the pupil has carried through successfully 
the learning experiences which have been designed to enable him 
to acquire the behavior involved in this step, a mastery test 
furnishes new situations in which he can demonstrate his com- 
mand of this behavior. A mastery test, hence, is quite dif- 
ferent from current achievement tests « In the first place, it 
is baped on the behavior to be learned in a sequential step in 
the learning system. There are as many mastery tests, needed as 
there are steps in the sequence employed. The complexity and/or 
difficulty of the behavior sampled by a mastery test is deter- 
mined by the complexity and/or difficulty of this behavior as it 
is to be used by the student as soon as he has mastered the step. 
For most school programs f the behavior learned in step 1 will 
then be used in step 2, and the behavior in step 2 will be used 
in step 3, and so on. This means that the requirement is for 
the student to be able to use in the next step the behavior he 
is learning in this step. This becomes the so-called 'briterion 
referenced norm." At the points in the ' sequence where the be- 
havior is primarily to be used outside the schoolroom or in 
other school subjects, the complexity and/or difficulty level 
commonly required become the "criterion referenced norms." This 
is another way of saying that a pupil has mastered the beliavior 
when he is able to employ it successfully in thd next stage of 
his life. 

Note that mastery tests are not designed ^to 
spread out a distribution of pupils. Much of the effor]t of the 
teacher and the pupils is to learn something and be able^ to use 
it, not primarily to say that one is at the 90th, 60th, 40th or 
some other percentile in the distribution of pupils' scores on 
an achievement test. When the student's performance on the 
mastery test . shows that he has acquired the behavior which the 
step was designed to help him learn, he moves on to the next 
step in the sequence. If his performance on the mastery test is 
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not satisf actory r a diagnostic test can be taken which enables 
him to identify the particular aspects of the behavior that he 
finds difficult. A diagnostic test is based on an analysis of 
the kinds of errors and difficulties found with some students. 
The test samples these more specific aspects of behavior to 
help identify the particular difficulty a student is having. If 
a diagnosis is to be helpful to the student ^ ha must also have 
available learning exercises that are designed to furnish more 
learning experiences for the particular aspects of behavior thxt 
he finds difficult and these learning experiences should invol^^ 
somewhat different approaches to the skilly the concept or the 
value from those employed in the lessons he worked on before. 

It is possible to summarize this discussion of 
improving instruction on the basis of evaluation by noting that 
where the instructional practices emphasize sorting and assume 
that the purpose of the school is to bring the relative per- 
formance of pupils in their school work in harmony with esti- 
mates of their scholastic aptitude or intelligence ^ norm-ref- 
erenced achievement tests are useful in identifying "under ach- 
ievers and indicating the persons who can be assigned to reme- 
dial courses or lower tracks ^ or given more time or more learn- 
ing exercises to improve their scores. Where the purpose of 
the school is to help each student to learn what the school 
seeks to teach ^ the instructional practices must provide for 
sequential learning and evaluation procedures for placement ^ 
mastery and diagnosis are important aids in improving instruc- 
tion. To develop the full potential of such a learning system 
requires a curriculum designed for sequential learning ^ appro- 
priate evaluation instruments^ and the development of classroom 
practices that enable teachers to manage such a system. 
Practices that help in this management process are pupil self- 
administered tests and learning exercises ^ peer-group teaching 
and learning; cross-age teaching and learning; the use of 
teacher aides for diagnosis and assignment of learning exercis- 
es. A number of schools are now pioneering in the development 
of improved and effective instructional procedures to help all 
children to learn. 
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IHE EVALUATION OF INSTRUCTORS— 
THE STATE OF THE ART AND THE ART OF THE STATE 



Michael Scriven 



(Michael Scriven is a Professor of Philosophy at the Univer- 
sity of California, Berkeley, is Chairnian of the Senate Committee on Teach- 
ing there, Chainian and designer of the Pkster Panel system used by NIE to 
evaluate all its RSD projects, and author of an absurd amount of writing 
about evaluation.) 



The state of the art is pitiful, a fact which lays i-nstructors 
open to artf^xL exploitation by the state in the form of regulations using, 
e.g., contact-hours as the sole measure of merit. 

Tliere are two very different problems involved in the evalu- 
ation of K-12 teachers (or State College faculty) in CJalifomia on the one 
liand, and University of California faculty on the other. The former can 
only be evaluated for competence/ incompetence or Pass/Not Pass, since raises 
and promotions are guaranteed: the only decision to be made is to fine or 
not to fine. The latter are supposedly also promoted/rewarded in a way 
that is connected with their teaching performance (as well as with their 
research, etc.). However, in the former systems initial appointments and 
tenure decisions may involve consideration of "merit beyond the minimum". 

It's much easier and/or more reliable to identify incompe- 
tence than superior performance, at the noment. In fact, we can do quite 
a good job at it. The paper presents some detailed examples of items and 
the relevant judges to support these claims, and then goes on to consider 
what can be done with regaxxi to estimting the quality of post-minimum, 
i.e., meritorious performance, both now and in the future. The general con- 
clusion is that we can now implement — although no one has yet done so — a 
systematic procedure that will give us considerable objectivity and worth- 
while reliability; that we can quickly move towards a position where we 
have very good reliability; and that failing to do either of these things 
will simply lead to further and well-deserved attempts at exploitation. 
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APPLE PIE, MOTHERHOOD AND EVALUATION 



Clarence L. Hall 



Clarence L. Hall is Adjunct Professor of 
Education at Claremont Graduate School, 
Claremont, California. He received both 
his B.A. and M. A. degrees from the 
University of Pacific and Ed. D. in 
Education Administration from Stanford 
University. 



Ten years ago I was absolutely convinced that the 
financial resources of this state and nation could support any level of 
educational endeavor. I was certain the dollars were available to fund 
our most grandiose strategies. I reasoned the problem was simply a 
matter of conflicting demands, domestic and foreign, with the 
investment in education always ending up as too low a priority. 

I continue to believe our societal priorities could stand 
some scrutiny, but now realize what was always true, that our 
resources are finite. This nation cannot afford a truly professional 
salary for a teacher for every twenty-five learners. We cannot afford 
thi3 array of all of the support personnel considered necessary in each 
of those classrooms. We cannot make available all educational 
technology, materials and equipment that's ours for the buying; nor can 
we house all learners in physical facilities that represent the best of 
contemporary functional design. We have to make some choices and 
set priorities. 

Even if the financial resources of tlie United States 
were unlimited, the Congress, the President, the California Legislature, 
the Governor and local communities all now add their increasing 
demands beyond the traditional impassioned pleas for more funds for 
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schools. Yes, we do have to make some choices and the various 
decision makers at all levels want to, and will, participate. Rightfully, 
they expect our reasoned arguments to be supported by evidence. 

We are indeed in the era of accountability in education 
and you who are evaluation experts are the new elite. Leon Lessinger, 
who has spoken and written more than anyone else about this movement, 
introduced the concepts of accountability and educational audit to the 
U. S. Office of Education. Prior to his acceptance of the position of 
Deputy Commissioner in Washington, D. C, as most of you know, Leon 
was a district superintendent in California. We were both members of 
a small consortium of superintendents who met regularly to exchange 
management and program ideas. I can recall discussing- -no debating-- 
the need for what he then called "quality control" in our schools. The 
issue is no longer debatable. 

Applicatiorr" of the "quid pro quo" principle to the 
requests of educational institutions is standard in the 70's. Federal, 
state and local policymakers expect no less than a statement of 
quantitative and qualitative results in understandable, meaningful terms. 
You, "the new elite, " bear a heavy burden in satisfying that expectation. 

Virtually every educational legislative proposal, state or 
federal, contains an evaluation component The last session of the 
California State Legislature, for example enacted SB 1302, the early 
childhood education bill, AB 2284, the bilingual programs bill, AB 1258 
the early childhood education program for American Indian children, 
and SB 90 providing $82,000,000 for educationally disadvantaged youth. 
Each of these programs requires evaluation of outcomes. Both the 
clarity and content of those evaluations will determine the maintenance, 
expansion or the discontinuance of funding for these new programs. 

Additionally, a host of existing programs are being 
questioned, scrutinized, analyzed and jeopardized by the probing of the 
Governor and his staff, the legislators and their administrative aides, 
and the Legislative Analyst and his staff. They want evidence- -and 
they are quite willing to invest funds to get the answers. At the same 
time that program proposals are being legitimately questioned, 
evaluation dollars are very easy, almost illegitimately easy, to come 
by. Evaluation has become an "American flag, motherhood, apple pie 
issue" in the minds of local policy makers, of the legislators and the 
Governor. 

Now Tm not an evaluation specialist, but a general 
practioner who has grown leery of apple pie issues. The easy math- 
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science money of the late 50*s and early 60's available for programs-- 
good, bad and worse --helped make me cautious. The easy language lab 
money, used poorly almost universally, helped make me cautious. We 
all recall how NDEA funds were there for the asking to install language 
labs that were on nobody's needs list, were neither imderstood, nor 
used properly and have since been dismantled. But the lab was in vogue. 
It was the thing to do, even if done badly. 

I feel a sense of responsibility to sound a warning about 
the "apple piedness'* of evaluation--to warn of the tendency to overact, 
to abuse, to overemphasize when issues are given noncritical sanction. 
Extremism flourishes on nonquestioning sanction. I simply want us to 
Ije certain we measure the right things with Hie right procedures in 
the proper context. 

By "proper context" I mean that the evaluation 
component should be an integral part of the comprehensive plan 
whether that plan applies to administrative decisions, instructional 
program decisions, or decisions about individual learners. That plan 
should include an assessment and prioritization of identified needs, the 
establishment of program objectives, generation of a range of 
activities, selection of the best from among the alternative courses of 
action, and development of a system for monitoring and reviewing the 
implementation of the plan. Good planning makes the design of good 
evaluation easier. Unfortunately, the demands and requirements for 
evaluation have too often come after the fact by imposition or after- 
thought. 

Secondly, I see a danger in reaching conclusions about 
programs based on oversimplified evaluation data. I have observed the 
evaluation of total school systems on the basis of the quality of the 
performing groups from the music department, the record of the 
football team and the reading scores. Dependence on measures of 
process and product outcomes for better educational decisions demands 
respectable measurement of the right variables. 

Albert Einstein defined education as "that which remains 
after youVe forgotten everything that you learned in school." Einstein 
possessed a talent for simplification of the complex into meaningful yet 
profound terms. He meant that education is not mastery of a body of 
content or sterile facts, but rather it is the acquisition of a thirst, an 
inquisitiveness, it^s an attitudinal set, it^s a style. It is the lasting 
influence the great teachers in our lives have had on us. Two 
exceptional teachers who touched my life were Miss Houlihan, a fourth 
grade teacher in Idaho and Dr. Ingego, a professor at University of 
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the Pacific, I can't tell you much/ if any, of the content of what they 
taught, but I could describe in detail the continuing influence they both 
had on me as a learner and human being. That influence is one of the 
very important things that remained after I forgot what I was taught in 
school — even important enough to assess, to measure, to evaluate. 

The uncertain tomorrows facing today's learners call for 
creativity, boldness, adaptability, compassion, and flexibility. There 
is a danger, however, that our systems of evaluation reward conformity, 
regurgitation and rigidity. The danger exists and thrives in those places 
where evaluation and measurement are defined as synonymous. Evaluation 
may include measurement, but the root of the word is "value." Therefore, 
to evaluate is to make a judgment of worth based on the relevant data. 

Dissertations completed last June at USC by House and 
Diamond studied sixteen educational goals for elementary children 
formulated by parents. The goals were then prioritized by teachers and 
parents. The goals rated 1, 2, 3, and 4 by parents and ranked 3, 4, 1, 
and 2 by teachers were: 1) to develop a positive attitude toward learning, 
2) to develop, a sense of honesty and fairness with respect toward other 
people, 3) to build a positive realistic self-image in order to accept the 
rewards of success and the consequences of failure, 4) to recognize 
one's self-worth and develop an awareness of his potential contribution 
to society. All the top four goals were decidedly in the affective domain. 
Reading, incidentally, was ranked sixth and eighth and math eighth and 
sixth by parents and teachers respectively. 

Assuming their findings are generalizable , one wonders 
how many schools have objectives and activities aimed at their top- 
ranked priorities and how or whether they are being evaluated-translated 
for staff and community. We can answer in the affirmative if: we can 
define the problems that need to be addressed, if we can identify the 
criteria used to determine that a problem exists, if we are willing to 
collect appropriate data , and if we are willing to accept indicators of 
change and not insist on sophisticated instruments and statistical 
treatments that tend to measure elephants with a micrometer when all 
we need to know is whether the elephant is bigger than a horse. 
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APPRAISING THE INSTRUCTIONAL PROCESS 
Madeline Hunter 



(Madeline Hunter is Principal of University Elementary 
School; Lecturer, School of Education; Director, Project 
"Linkage, " at the University of California, Los Angeles, 
She received her B. S, , B, A, , M. Ed. , and Doctorate 
degrees from the University of California, Los Angeles.) 

Clouds of mysticism have shrouded the process of instruction, 
obscuring cause- effect relationships. Attempts to pierce this veil have been 
resisted by some who would retain the mystery and rely on intuition to guide 
instruction. Others of us, impatient with unnecessary learning failures, are 
seeking ways to bring predictable learning success under our span of instructional 
control. As a result, a Teacher Appraisal Instrument (T, A, I. ) has been developed 
which makes successful learning predictable and successful teaching explainable. 

Two Ijasic generalizations, free from informed contradiction, guided 
our search. These generalizations constitute invarient principles which are 
applicable to all learnmg situations regardless of content, the learner's age, 
previous experience, ethnic or socioeconomic derivation- 

The first generalization is related to the incremental nature of learning. 
Learnings are built one on the other with basic learnings supporting and making 
possible more complex learnings. It is impossible for a learner to achieve a 
higher order learning, without also having achieved the subordinate learnings which 
support it. 

The second generalization which focused our efforts, is related to the 
factors affecting learning which are accepted and validated as basic principles by 
all learning theorists regardless of their particular concept^ial orientation. These 
factors respond to instructional manipulation and affect a student's motivation to 
learn, the rate and degree of his learning, his retention of that learning, and his 
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ability to transfer that learning to iiew situations where it is applicable. While 
these factors may take different form with individual learners, as principles they 
are invarient to all learners. 

As we at UCLA began to apply these two basic generalizations to the 
appraisal of the instructional process, a third encompassing insight into the 
t-^aching- learning process emerged as critical. We learned that time is the coin 
of teaching and it is expended to "purchase" learning. Like all currency it can be 
expended wisely or frittered away with nothing to show for its use. Wise invest- 
ment of instructional time to produce efficient and effective learning is determined 
by the valid implementation of the two basic generalizations. Wasteful squandering 
of instructional time is the result of actions which are in violation of these two 
generalizations. Consequently, any evaluation of the instructional process must 
be based on the investment of the learner's time (whether such investment is 
consonant or dissonant with current knowledge related to human learning), in terms 
of the following questions: 

1. Is the instructional process proceeding toward a perceivable objec- 
tive, or is it a meandering path where time is dissipated without appropriate 
learning gain? Additional learnings which are complementary to the target learning 
are encompassed in the term "appropriate, " but learnings that are interfering, 
tangential or antithetical to the objective are deemed inappropriate. In this way, 
learning time is focused and effectively used rather than being happenstance, 
random or diffused with little or no desirable learning return for the time and 
effort of student and teacher. 

A positive answer to this first question in no way eliminates creativity 
or imposes rigidity . If a tangential or nonrelated learning objective emerges from 
the student, the original objective may be altered to accommodate it or the tangen- 
tial learning may be referred to a future instructional episode. 

2. Is the instructional objective at the right level of difficulty for the 
learners who are investing time ? This implies that the particular learning step 
being taken toward the objective is an achievable one by these learner8--not an 
objective that is so difficult its achievement is impossible or one so easy it requires 
no learning effort or it has already been achieved. 

3. Is there constant monitoring of the degree of achievement of the 
objective so redundance or acceleration can be built into the instructional process 
if either is indicated? "Dip sticking" is the term which indicates that "soundings" 
are taken at frequent intervals to validate learning achievement before moving 
ahead as well as to avoid investing time on a learning that already has been accom- 
plished. 
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These first three questions are related to content--the 'Vhat" of 

learning. 

The next two questions used to appraise the instructional process 
involve the "how" of learning, or the congruence of the learners activity and effort 
to principles which research ha^s demonstrated to be facilitating or accelerating to 
learning. For convenience, these principles have been categorized into four groups 
(1) those principles that affect the learner's motivation; (2) those that affect his 
rate and degree of learning; (3) those that influence his retention of what he had 
learned; and (4) those that contribute to his ability to transfer the learning he 
achieved to new situations where that learning is applicable. 

Based on these categories of learning principles, the fourth and fifth 
appraisal questions are asked: 

4. In which ways are the time and energy expended by learner and 
teacher consonant with principles of efficient and effective learning? 

5. Is there dissonance between time and energy expended and princi- 
pies of learning? If so, which principles are being violaited ? 

The T. A. L is similar to the evaluation of the nutrient qualities of food 
regardless of the particular menu or the way it is served, for any appraisal of the 
instructional process must focus on the invarient "nutrients" which promote human 
learning. 

As a result, the T. A. I. is applicable regardless of whether a teacher 
is working with one leaimer, with a small group or with a total classroom, to any 
instructional process when a learner is working by himself, with a friend, with an 
instructional module, programed instruction, textbook or worksheet, to the self- 
contained classroom, team teaching, open structure, individualized instruction, 
nongraded or whatever organizational program is in effect. 

The T. A. L is only in its infancy, but it is a robust infant that has 
promise of growing into a curriculum guide for the preparation of teachers, a 
diagnostic instrument to direct staff development and a valid instrument for 
evaluation of professional performance in the classroom. 
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GRUBBING FOR INSTRUCTIONAIi IMPROVEMENT TRUFFLES 

Eva L* Baker 



Eva L. Baker is an Associate Professor of 
Learning and Instruction at the University 
of Calif ornia, Los Angeles* She received 
her B.A., M^A., and Doctorate degrees from 
the University of California, Los Angeles. 



The credibility of the schools rests in part on the public* s 
assumption that teachers know hov to teach and that schools are itmportant 
places for children to be. Reflect for a moment on the state of the educa- 
tional world if these propositions were not widely held. 

The schools and the people who work in them may be facing an 
exciting but potentially risky test during the nerb few years. The issue 
that will race before ois is mildly named "instructional improvement." In- 
structional improvement has been a tone chimed by in-service coordinators as 
effortlessly and predictably as the noon bells at UCLA. The music may now be 
changing. 

Instructional improvement implies at least the following set 

of conditions. 

1. Instruction (defined as you wish) exists. 

2. Instruction will recur. 

3. There is some basis for expecting improvement. 

4. There is a way to determine if improvement has been made. 

The reason that instructional improvement will become a 
serious issue relates to points 3 and h above. Until recently, bases for 
anticipating and determining instructional improvement were not widely imple- 
mented in our schools. We have learned these last few years another term for 
these activities: evalxiation. 

A flashback to instructional improvement practices mi^t be 
helpful at this point to promote recognition of why we are in such a dramatic 
time. In previous epochs, teacher behavior was assessed by judgments or 
rating of process. A teacher mi^it be given an overall evaluation as excel- 
lent or poor by his or her supervisor. The criteria for such evaluations 
may have been less specific than some desired. However, when the criteria 
were articulated, as in some teacher evaD^uation scales developed for use with 
student teachers, the dimensions selected were often only tenuo\isly related 
to the act of teaching. Other approaches to the assessment of teachers have 
been more specifically geared to instruction. For instamce, a lesson mliSit 
be taught, and then the supervisor might discuss with the teacher particular 
aspects that seemed to need improvement. The teacher's lecture style or 
organization might be identified as deficient or a discussion session mi^t 
be criticized because it did not involve all participants. The problem with 
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this approach to instructional improvement should he clear on the basis of 
experience. It is stark vhen researcih evidence is consxiltec. The single, 
undisputahle fact of life is that there is no evidence to support that a 
given instructional method will unalterably succeed vith all learners in all 
tasks with all teachers. There etren^t efven methods that have reasonable 
probabilities of success in very limited aitxiations. Thus, all the attention 
directed to process and tise of methods in the absence of suggestive, let 
alone compelling data, probably contributed very little in terms of improving 
the instruction that our children received. 

The critical linkage, of co\irse, is the relationship between 
any instructional improvement recommendation and its likelihood of modifying 
student performance. If instructional methods have not proved fruitful in 
terms of student performance, lines of research have identified certain in- 
stiuctional variables that are positively related to learner achievement. In 
simple terms, this means that teachers who tend to use given vfitriables or 
principles in their instruction tend to have students perform significantly 
better. Variables are method- free. They can be used in different tasks by 
teachers with different personal styles. Barak Rosenshine and Norma Purst 
reported on eleven such variables that were positively related to achievement 
in a number of independent research studies. Some observed variables like 
"clarity" suffer because the ratings were highly inferential. A teacher was 
judged to be "clear" in presentation, but specific behaviors or criteria for 
clarity were not described. Thus, training someone to be "clefiU"" was not all 
that easy. Rosenshine and Furst identified a small set of instructionel 
variables that could be observed directly with only a minimum of inference. 
These variables or principles could be taught to teachers with only minor 
problems in translation from the research setting. Two of these variables 
had very strong support from resesurch: prorviding students the opportunity to 
learn criterion performance and task orientation of teacher. 

Work at UCIA and at the University of Illinois may have ex- ^ 
tended the list of powerful principles. In a series of studies at UCIA the 
following principles were significantly correlated with short term measures 
of student achievement: 

Practice (opportunity to leaxn criterion material). 

Prompting (reducing the cues that sequence practice). 

Individualization (attempting to differentiate instruction for sub- 
groups of learners). 

Knowledge of results (Informing the learner of the adequacy of responses). 
Of considerable interest to me is that certain variables also 
correlated significemtly with the interest levels of the learners. Princi- 
ples that were significantly related to interest rating,^ were: 

Practice 

Knowledge of results 

Motivation (attempting to eacplain or Induce a reason for the instruc- 
tional activity) 

Yet, the move toward accountability and serious evaluation of 
the performance of students must encourage us to make use of research of this 
sort, even at its present, inconcltuaive level. Perhaps, we can head off the 
frij^jtening prospect that may await us in the sdiools: the specter of poor 
adiievement cmd no way of remedying the situation. 
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THE NEW STATE ASSESSMENT PROGRAM; BOON 



OR BOOMARANG 



Edwin P. Larsen 



CEdwin P. Larsen is Director of Research for the 
Oakland Public Schools. He received his B.A. 
and M.A. degrees from the University of California, 
Berkeley.) 

\ DECADE OF DEBATE AND COMPLAINTS ( 

Since the first State Assessment Act sponsored by Winton in 
1961 went into effect, millions of dollars and millions of hours of 
teachers' and students' time have been invested in administering stan- 
dardized achievement tests to elementary and secondary school youngsters. 
School practitioners, and in recent years, some cciniaunity spokesmen have 
complained vociferously regarding many aspects of the state testing 
mandatet^. However, legislature has held tenaciously to their demand for 
evaluative feedback on the effectiveness of California's multi-billion 
dollar educational program. 

I SOME SIGNS OF PROGRESS | 

After a decade of vociferous, and sometimes bitter, confron- 
tation, criticism and debate a new "master plan" for statewide assessment 
has been formulated. The most apparent changes which are being discussed 
are: a clear swing to an assessment program designed to meet state needs 
only; a reduction in time and dollar costs co districts so that the local 
agencies can implement their own assessment plans; abolition of required 
"I.Q." testing; use of tests design for "California Objectives;" schools 
and districts are to be the units of measure as opposed to the individual 
as the result of the matrix sampling approach (there will be no student 
scores). 

As a member of the district-level cadre which has been very 
critical of antecedent state assessment programs, tho author believes that 
some highly significant "context," or process, changes have also occurred. 
For the first time, the legislature has recruited a broad spectrum of 
knowledgeable people in the area of measurement to help design the state- 
wide assessuient program. 

Also for the first time, individuals and groups from the 
professional ranks have perservered beyond the level of critical rhetoric; 
that is, some positive suggestions for alternative assessment strategies 
have been 5et forth. 
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i SOME CAUTIONS AND SOME PROBLEMS | 

There are hopes of moving statewide assessment from the level 
IL^^.T ./y^f instrument for improving the educational oppor^ 

tunities for C'llifornia's youth. However, we have not arrived at that point 
yet. Following are some issues and concerns which still persist. 

, , , _ .^e Entry Level Test is, perhaps, the same old I.Q. goal in 
./.f '-^^ traditional terms, will be generated 

for individuals, schools, or districts. However, it seems that the new 
assessment plan will rely heavily on the old, simplistic "aptitude-achieve- 
ment prediction model, if the entry level test is, in fact, viewed as an 
nJ^ Jufi. ""^^^ ^^^^ P^^^^ There is the distinct 

Spn?f i '^f ''^''''^^ ^^^^ ^^"^ ^^^^ phrases, such as "These stu- 

dents have perfo^ed on achievement measures 'as well as may be expected'.'' 
This can constitute the most insidious type of low expectancy syndrome of all 

The reliability of the data base (scores) should continue to 
be a matter of concern. Those involved with the mass state asL^essment 
programs of the 60 's know of the variety of problems which have arisen in 
the poor administration of the tests, the incomplete reporting of scores, 
and a number of other obstructionistic devices which have plagued the state 
testing program from time to time. The large proportion of unexplained 
variance seen in the correlational analyses undoubtedly grows not only from 
the nature and refinement of the predictor (independent) variables but also 
a substantial error factor in the predicted (dependent) variables. The 
existance of hundreds of thousands of pieces of data is no real substitute 
for reliable input into research studies and this may continue to prevent 
the discovery of meaningful and useful information in the state assessment 
program* The only answer to some of these problems will be continuing 
efforts to win general acceptance of the programs. 

n.oar.« " "^^^^f^"^^^ the data generated by the new assessment 

program may or may not be greater than in previous programs. The follv of 

mass regression studies has been demonstrated well in recent years. From 
a practitioners point of view there have been no substantial or productive 
inaights in tc the effectiveness of various program and/or policy alterna- 
tives growing out studies at the State or National levels. The factors 
which have been shown to account for the variability in classical achieve- 
ment indexes are not those over which the educational institutions have, 
or hope to, have no direct control. 

Some new technologies or some better application of existing 
technologies will have to be brought to bear. The power of the computer is 
no substitute for good research methods. The State Department of Education 
has proposed using standard regression model approaches to identify clearly 
discrepant (high or low) performance and then to follow this with in d^pth 
investigations to verify the reliability of these observations and to 
illuminate the practices or factors which appear to contribute to substan- 
tiated variations in achievement performance. Perhaps this may yield some 
information useful to decision-makers at state and local levels. If thirS 
does not occur, schools throughout the state will continue to be plagued, 
rather than assisted, by the state assessment prograts. 
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NEW DIRECTIONS FOR THE CALIFORNIA ASSESSMENT PROGRAM 



Dale Carlson 



(Dale Carlson is Assistant Chief, Office o'l 
Program Evaluation and Research, California 
State Department of Education. He received 
his B.S. degree from St. Cloud State College, 
his M.A. from Ohio State, and the Ph.D. from 
the University of Southern California.) 

Although a number of modifications have been made to the 
California statewide testing program since its inception in 1961, the most 
far-reaching revisions are now being mad'3 as a result of AB 665 of 1972. 
In order to properly understand the direction and dimensions, of this pro* 
gram, it is essential that the underlying assumptions about the assessment 
program and its purposes be considered. Seven key postulates are presented 
below. 

1. The pr-fmary purpose of any testing program (or evalu- 
ation system) is to provide information to assist in decision-making. 

2. The variety of types of decisions to be made demand that 
information be collected and reported at various levels of specificity (in 
terms of conterit or objectives) and various levels of aggregation (in terms 
of the data unit; pupil, school, district or state). Obviously, the infor- 
mation which a teacher needs to make decisions related to specific pupils 
would be of little use to a legislator, and vice versa. 

^ 3. To be useful for program improvement, the results of an 
assessment must be reported for each of the accepted program objectives. 

4. To arrive at the most complete and accurate evaluation of 
a program, the results must be presented in both absolute and relative terms, 
i.e., they must be both criterion referenced and normative-referenced. The 
"actual" or "real" performance of the pupils must be presented as well as a 
description of that performance in comparison to (1) a general population, 
like all other schools, and (2) to a specific reference group, like all 
similar schools. 

5. A single assessment program designed to serve the needs 
of all consximers of pupil achievement data would be restrictive and inef* 
ficlent. For example, a highly differentiated criterion- referenced assess- 
ment system mandated for use with all pupils in all districts in the state 
would be excessively expensive and would probably be inappropriate for many 
districts. 
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6. The main function of a statewide assessment program 
should be to provide information to asoist in making decisions with state- 
wide impact* This information should be gathered most economically, 
absorbing a minimum of instructior I time for testing. Since the informa- 
tion is for use primarily by the s^ate, the cost ought to be borne pri* 
marily by the state. The district and school would then be free and re- 
sponsible to develop assessment programs which provide information for 
making all decisions at the district, school and classroom levels*. 

7. There is sufficient justification for an assessment 
program which is focused on providing information for state level purposes. 
Such a program must be designed to answer the following types of questions. 

• On the whole, how are California pupils progressing in 
mastering the basic and advanced educational skills and learnings? 

• Should education receive a gre^'iter or lesser proportion 
of the state's resources, in comparison to welfare or ecology? 

• What are the areas of greatest need in California 
education, e , g, , reading, math, drug abuse education? 

• Which districts and schools have particular needs for 
additional assistance or resources? 

• What are the characteristics of unusually effective 
programs? How can information about those programs be used to improve 
other programs? 

• What is the statewide impact of special categorical 
programs and projects, i.e,, the Miller-Unruh Program? 

With this background, the dimensions of the revised program 
can be explained more rationally and evaluated more fairly. The major 
features are outlined below. 

1. The test administered io all pupils In grade one is no 
longer a reading achievement test. Rather, it is an assessment of readiness 
lor typical school instruction. It is a relatively short test which is 
very easy for most children. Its purpose is to provide information on the 
entry level skills of groups of pupils. This information will be used to 
identify similar schools and districts, so that test results from grades 
two and three can be interpreted in relation to the entry level of the 
pupils. 

2. Testing in grades two and three (reading) and in grades 
six and twelve (reading, math, spelling and effectiveness of written ex- 
pression) will be tested on a matrix sampling basis. All pupils will con 
tinue to be tested, but each pupil will take a very short subtest, repre- 
senting only a portion of all questions on the full test. This type of 
testing yields information on a wider array of instructional objectives 
while decreasing the amount of testing time. 

The spring of 1974 will see the implementation of this pro- 
gram in grades two and three. Grades six and twelve tests will be 
developed for use in 1974-75. 

3. A substantial portion of the financial burden of the 
program will be shifted from the districts to the state. All costs of the 
grade one program will be borne by the state, as will the costs for all 
test materials at the other grade levels. 

4. Test results will only be calculated and reported for 
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groups of pupils, i.e., for schools and districts. This method yields 
reliable data for all primary purposes of the program without interfering 
with local assessment programs designed for pupil diagnosis or teacher 
evaluation^ 

5. It is an objectives-based program since common sets of 
instructional objectives have been developed. Only those objectives will 
be assessed and the results will be reported according to those objectives. 

6. Test results for schools and districts will be reported 
in terms of the average percent of pupils answering a set of questions 
correctly. This information will be analyzed and interpreted in comparison 
(I) to all schools and districts and (2) to schools and districts with 
similar characteristics. Operation/ally, this last type of comparison will 
probably continue to be done with the help of multiple regression analysis 
whereby the mean for a school or district is compared to its predicted mean. 
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COMMUNITY INVOLVEMENT IN 



EVALUATION AND INSTRUCTION 



Henry M. Mestre, Jr. 

(Henry M. MeBtre, Jr. is Deputjf 
Director for Community Services 
of the Spa ni sh Speak ing Un it y Coun- 
cil of Alarreda County- He also 
serves as Chairman of the Task on 
Curriculum and Instruct ion ^nd 
Continuing Moderator of the Coor- 
dinating Council of Oakland 
Public Schools* Master Plan Citizeni? 
Cnmrr^ittco. He received his BA from 
the University of California, 
Berkeley . ) 

When I firs^ learned of the topic assigned to 
this panel, I rather welcomed the opportunity to share i>ome 
of my experiences as a lay participant m the educational pro- 
cess. However, my initial elation gave 'ay to feel ngs akin 
to those Daniel might have experienced wht.m he entered the lions' 
den. The topic itself, "Ccrrjr.un 1 1 y Invo Ivenr.ent in Evaluation and 
Instruction," was the prmcipuil source of my misgivings. The 
use of the word "comrr^unity" -^.ets educators in one cam^p and the 
"comniunity" in another, hir,ting at the adversary situation that 
is frequently present in sc hool- commun i t y relations. An exam- 
ination of my experiences in this area added to my concern and 
confirmed the existence of a barrier between educators and the 
communities they serve; a wall whcye stones include fear and 
cultural differences, but whose keystone is a myth propagated 
by educators and generally accepted by the lay public. That 
njyth simply stated is that the lay public does not have the skill 
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or knowledge to presume to tell educators how to do their Dob 
or to do their job. 

However, the increase? i cost of educat ion and its 
decreasing ef f ect ivene?^ s , admittedly measured by crude eval- 
uative inst rum.ent s , have led growing nuiri:>ers of parents and 
other lay people to seriously question what goes on in the 
classroom and to demand more meaningful roles in those activ- 
ities. Why? Because schools don't exist in a vacuusi. They 
are an integral part of the community: preparing its citizens, 
particularly the young, to function effectively in the commu- 
nity and to participate fully in its democratic processes. 
Beyojjd th^se communal goals, schools also have a responsibility 
to the individual; building character on the foundation of 
values laid by tl.c family. 

These practical and moral im.peratives have pro- 
pelled an assault on the barrier between schools and their 
comm.uni t ies . Federal, state and, in some instances, local 
statutes and policy have provided the vehicles for the assault. 
Over the past few years, parent and community advisory groups 
have increased greatly along with the use of lay instructional 
assistants. Today, many educators and members of the lay 
comjr.unity find themselves in terra incoqn i t a exploring the 
limits of a new relationship. 

Returning to what I have called the keystone myth 
we find that educational skills and knowledge do have a bearing 
on the limits of effective community participation, but not as 
an inviolable or exclusionary barrier. The real limit comes 
from the capacity of th^ lay public to acquire the skill and 
knowledge necessary fur meaningful participation in the educa- 
tional process: wnether it be in evaluation and instruction or 
in policy making and planning. 

Naturally, this point of view places an added 
responsibility on the educator: to provide the lay comn*unity 
with a better understanding of the science of learning. Is that 
science so arcane that the lay public cannot be involved in its 
practice without extensive preparation? Newer forms of teacher 
training such as Teacher Corps seem to indicate that it i£:n*t. 
If a college senior with ziiinimal training can begin practicing 
the pro f ess ion , why can • t a lay person , whose life exper iences 
are much broader, play any equally imp>ortant role in the 
classroom or in the evaluation of classroom activities? 
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There are 3 variety of experiences in daily life 
that give the averaqe lay person a sound foundation on which 
to build more precise knowledge of riucation. Anyon^-' who has 
ever worked sooner or later has had the experience of •'breaking 
in- a new errployee. Anyone who plays has had the experience of 
teaching a newcoir.er the rules of the game. 

To get a better sense of this concept, let's take 
a parent as an exan.pl e and eavesdrop on an iiriaginary, but not 
iirprobable conver sat ion : 

Parent: "How old are you? 

c Cuantos anos tienes, 

Juan itc?- 
Child. "Three, :tres: (Holding 

up three fingers) . 
Parent: ''TMuy bienl (Hugging 

the child) . ^ 

In this brief encounter we find excellent examples of bilingual 
instruction and positive reinforcement. The parent doesn't 
know these names, but has an instinctive understanding of the 
learning process. In every sense of the word parents are the 
child's first teachers and their instruction isn't limited to 
toilet training. They are the ones who create the environment 
in which the child learns to walk, to talk and the rudiments of 
rational thought. 

It is the respons ibi 1 ity of the educator to 
recognize and respect these and other experiences the interested 
lay man or woman brings to the school. This element has been 
present in every successful school-* coiiarunity activity with 
which I have been associated. The second ingredient has been 
timing. The educators responsible for these projects have 
built in the lead time necessary to bring everyone up to a point 
where they are approximately the same level. They have also 
provided the participants with printed background material: very 
often material they have prepared themselves specifically for 
this purpose. 

Certainly, the few points made here don't answer 
all the problems encountered in the area of comn^unity involve- 
ment. Most educator^ if they are at all receptive to the idea 
of community involvement, seemed to be more concerned with the 
num±>er of paiticipants than with the quality of participation. 
If you plan a meaningful role for the lay participant and the 
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experience gratifying, attendance isn't really a problem. 
Developing a new rr.ental set about the ability of the lay commu- 
nity to absorb the knowledge necessary for effective participa- 
tion in learning *3ctivities is a giant step toward a successful 
experience. Hopefully, the accumulation of successful exper- 
iences will bring down the wall that separates the school from 
the comn^unity it serves. 
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COMMUNITY IN\mVEMENT IN EDUCATIONAL DECISION-MAKING: 



AN EMERGING ROLE 



Keith Echeverri 



(Keith Echeverri is Chief Consultant 
to the Joint Coinmittee on Educational 
Goals and Evaluation of the Calif- 
ornia Legislature. He received his 
B.A. degree from Pomona College and 
his M.A. from Claremont Graduate 
School and University Center, 
where he was engaged in doctoral 
studies before going to Sacramento.) 

Whether and how the community should be involved 
in educational decision-making depends on the function of the 
school as a social institution. 

Why do we have schools? 

The answer to that question will disclose the 
answers to the previous questions concerning the role of the 
community in education. 

It seems to me that we are torn between the idea 
that the school is a place where children are socialized and 
the idea that school is a place where children seek out 
personal discoveries and meanings. 

Many people may see those two goals as mutually 
exclusive or mutually supp>ortive; the position depends on one's 
view of the nature of man. If man is inherently evil, he must 
be taught to live without inflicting harm on others or be 
punished. If man's natural instincts lead him to get along, to 
accept others and to love, then personal freedom need not be 
feared or taken away. 

The point is this: if schools are primarily 
socializing institutions, where literacy, social adjustment, 
and job placement are the important goals, then the role of the 
community in educational decision-making should be one of 
perpetuating the status quo, establishing performance standards 
for students and school personnel, providing basic maintenance 
systems, and generally protecting and preserving law and order. 
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If, on the other hand, the schools are viewed 
primarily as human growth centers, where self-esteem, intellec- 
tual and creative curiosity, and self-actualization are the 
important goals, then democratic principles should be incorpor- 
ated within the context of a dynamic, changing, person-centered 
organization. In this environment, the resources of all 
school= community members would be valued, and their participa- 
tion would be meaningful. 

The question may be whether we want to nurture 
followership or leadership. At different times we all should 
have the capacity for both; the question is one of emphasis. 

Aside from this central issue of the purpose of 
schools in relation to society, there are other than functional 
reasons for community involvement which few school boards and 
administrators can ignore. 

Political support for schools is necessary to 
gain revenues, maintain programs, and retain personnel. 

Public relations must be positive and supportive 
to assure that schools project a popular image within the 
community. 

Finally, there are moral and phxlosophic bases 
for community participation in a democratic society which 
claims to value individual rights and responsibilities. 

Given all the reasons for including community 
members in educational decision-making, the strongest argument 
rests on one's view of the purpose of schooling. 

If students go to school to be told how to 
behave and what to know, then the community need not become 
really involved, because they've already been told (that's why 
the "new Math" is such a problem) . 

However, if students go to school to learn about 
themselves and develop their own perceptions of the world, then 
parents need to be actively involved in learning and growing 
with their children, so that they will be able to relate and 
communicate and share perceptions with their children. 

Growing together — parent and child, teacher 
and student, school and community — would be the mark of a 
good educational process. 

Recently a brain-storming session was held in 
Sacramento to develop ir'^eas for improving the chances that 
community people will be meaningfully involved in educational 
policy decisions. I think you will find the results interesting 

4 
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especially since the contributors hold important positions in 
such established California groups and agencies as PTA, League 
of Women Voters, Women For, NAACP, the Department of Education, 
County School Offices, and the Legislature. 

The results are presented both in sxammary and as 
a list of the best individual ideas. The sxammary statement was 
developed and supported by the contributors. 

SUMMARY OF MEETING 

Question asked: "What steps can be taken to improve the 
chances that community people will be meaningfully 
involved in educational policy decisions at all levels?" 

Summary: 

I. To achieve meaningful community involvement we believe 
school districts should be required to have elected 
school=community committees at each school with certain 
delegated responsibilities to work with administrators and 
staff in budget, personnel and program decisions. 

II. Legislation developed to accomplish the above must 
include a well defined vehicle for delineating responsi- 
bility between the District Board and the local school= 
communities . 

A. Training programs must be established for adminis- 
trators, staff and community in all that is involved 
in productive participatory management 

1. Hire community people as consultants 

2. Outside analysis of decision-making process 

B. Have an ombudsman for the citizenry (on the Board 
payroll) with full authority to seek information 
anywhere in the school system, reporting to the 
citizenry as well as the Board. 

INDIVIDUAL IDEAS 

Related to Boards of Education ; 

1. Elect responsive school boards 

2. Have an ombudsman for the citizenry on the Board 
payroll, with full authority to seek information anywhere 
in the s choo 1 sy s t em > 

3. Student involvement on Boards of Education 
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4. Delineate authority of Boards as elected representa- 
tives of the people 

5. The Board and administration should accept feasible 
community recommendations 

Related to training ; 

6. Select and train administrators to work with the 
community (interpersonal relations) 

7. Provide trained discussion leaders and recorders at 
meetings 

8. Increase awareness of model programs (alternatives) 

9. Locate and publicize successful models of responsive 
government 

10. Community representatives will need training 

R elated to schoolgcommunity committees ; 

11* Mandate a share of decision-making beyond advisory 
capacity of councils 

12. Set out clearly the opportunities and the limits for 
decision-making. Establish and publicize a contract. 
Don't make promises that can't be kept. Define advisory 
and authority . 

13. In many cases, a parental pressure or advocacy group 
is necessary - 

a. Beat on the doors;- demand a share of decision- 
making 

b. "Sell" community ideas to Board 

14. Ad hoc community groups formed to make recommenda- 
tions to the Board instead of to the administration 

15. Elect school committees with certain delegated 
responsibilities to work in cooperation with administration 
in budget, personnel and program decisions 

Other ideas : 

16. Parent involvement as classroom aids 

17. We must first describe why community involvement is 
important — roles will come from that 
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18. Administrative reports at public meetings on 
educational practices and progress 

19. Hold hearings, debates on policy issues — on 
neutral premises — with administrators present as 
witnesses or participants 

20. Outside analysis of district decision-making 
processes 



Most of the conclusions reached at the meeting 
point toward legisl ati ve mandates of new public governance 
structures. It will be interesting to note whether or not 
other means/methods of achieving effective community involve- 
ment are tried, and by whom, and where. Educational leadership 
will be in viev; as we watch. We may even choose to lead. 
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RENOVATING INEFFECTUAL EVALUATION PROGRAMS 



Chairman: Dr. Lester Ristow 

Retired 



Participants: Marvin Alkin, Director 

Center for Study of Evaluation 
UCLA 

Robert C. Otto, Assistant Superintendent 
Research and Development 
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REORGANIZING SCHOOL DISTRICT 



EVALUATION DEPARIMENTS 

Marvin C. Alkin 
Robert C. Otto 



(Marvin C. Alkin is Professor of Education 
and EKrector of the Center for the Study of 
Evaluation at the University of California, 
Los Angeles. He received his A. B. and 

A. degrees from San Jose State University 
and his Doctorate from Stanford University. 
Robert C. Otto is Assistant Superintendent, 
Research and Development, for the Gross- 
mont Union High School District. He holds 
a B. S. from the University of Nebraska, 
M. S. from Creighton University, M* A. from 
Chapman College, and Doctorate from the 
University of California, Los Angeles. ) 

Organized evaluation activities in most school 
districts just simply '*growed like Topsy. '* And, their location within 
organizations and the way in which evaluation is conducted is evidence 
of this. The increase in testing in school districts, requirements for 
evaluation of Federally funded programs, needs assessment in California, 
and other evaluation activities have been introduced at different times, 
shunted to available district personnel, and frequently find themselves 
in disparate parts of the same organization. 

These are not surprising findings. The nature 
of a bureaucracy dictates that management ctart with the program that 
exists and simply add on new functions where they "fit best. Freqaently 
this leads to organizational structures that lack rationale, are inefficient 
and improperly staffed and only make logical sense to insiders because 
they carry the brand of familiarity. 
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Some school (' itricts have evali-&.*ion departmentf 
that are merely extensions of the exvflting testing unit aiid whose per- 
sonnel, while they know tests, are insensitive to the procedures necessary 
in formative or summative evaluation. Some school districts charge 
the "Federal project director" with evaluation r e sponsihilitie s seeming 
to say that only Federal projects need evaluation (which is all too fre- 
quently the practice). Many school districts have charged a curriculum 
coordinator or other surh district office personnel with the responsibility 
for Stat e- required needs assessment, without recognizing the obvious 
tie-in of this procedure to the establishment of a district- wide evaluation 
and testing program. 

We would maintain that it is time for school districts 
to reconsider the role of evaluation. It is time to consider a rational 
evaluation role unencumbered by existinje allocations of responsibilities. 
It is time to try to rnaVe sense of the role and function of evaluation in 
school districts. In this oral presentation the authors will present some 
thoughts on reorganizing school district evaluation units. It is time. 
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r.VA!.l'ATIN(. AM> IMTFOVING THE OVALITY 
OF INSTKrCTlON IN HIGHf.R EDUCATION 



Cha i rman : 



Dr. TzTA Vveth 

Professor, Psychological Found at ions 

of Education L>€'par ttnent 
School of Education 

California State University, Northridge 



Par 1 1 clpant s ; 



Robert ( . Wi 1 son 

Teaching Innovation and Evaluation Services 
University of California, Berkeley 



Lynn Wood 

Assistant Director, Teaching Innovation and 

Evaluation Services 
University of California, Berkeley 
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EVALUATING COLLEGE TEACHING ON THE BASIS OF INFORMATION NEEDS 
FOR WHAT? BY WHOM? AiND WHAT FOR? 



Robert C. Wilson and Lynn Wood 

(Robert C. Wilson is Director of Teaching 
Innovation and Evaluation Services (TIES) 
at the University of California, Berkeley, 
Lynn Wood Is Assistant Director of TIES,) 

Historically there have been three primary purposes for 
evaluating instructors and courses at the college and university level. 
Each of the purposes has corresponded to the information needs of a dif- 
ferent segment of the academic world. Students have desired such eval- 
uations primarily to assist them in making choices among courses and 
instructors. Faculty review committees and college administrators desire 
such evaluations to give them the kinds of documentation of teaching 
effectiveness which are increasingly required in making promotion and 
tenure decisions. Individual faculty members require various kinds of 
''feedback** to assess their own effectiveness as teachers and to make in- 
formed decisions about desirable chdnges in their courses and methods 
of instruction. 

One way of de/ining eval uat ion is: the gathering of in- 
formation relevant to one*s needs to make decisior^s, making Judgments 
about that information, and making decisions on the basis of those 
judgments. When the nature of the decisions to be made differs, informa- 
tion needs can be also expected to differ. That is not to say that some 
of the same information about courses and instructors might not be use- 
ful to students, administrators, and individual faculty members; it is 
unlikely, however, that a single set of information or a single evaluation 
procedure will best serve the needs of these three groups. 

If the evaluation model suggested here is felt to be a 
valid one, It follows that efforts to initiate a successful, i.e. useful 
program of teaching evaluation require careful preliminary study. The 
first question to be addressed is: For what purpose is the evaluation 
being undertaken, i.e., what kinds of decisions are expected to result 
from the evaluation? Who will make those decisions? What kinds of in- 
formation does That group feel that it needs in order to make those 
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decisions? Once these major questions have been answered, subsequent 
decisions about the ev.^luation procedure itself may be somewhat easier 
to make because, in part, they will be dictated by the information needs 
of the decision-making group. Some of the parameters of teaching eval- 
uation which might be expected to vary in interest and importance to the 
three decision-making groups include: 

a) the scope of information (e.g., all instructors? all 
courses? every term?) 

b) the source of information (e.g., present students? 
former studi^nts? colleagues? the instructor himself?) 

c) the kind of information (e.g., data on the instructor's 
ob'^ctives? communication skills? presentation styles? 
accessibility to students? data on the course format? 
assignments? grading? data on student's motivations? 
interest and enjoyment? learning gains?) 

d) the method of gathering information (e.g., end-of-course 
questionnaire? interviews? pre-and post-tests of student 
change? colleague visitations? videotape and interaction 
analys i s?) 

e) the specificity of information (e.g., broad descriptors 
known to be generally related to teaching excellence? 
detailed items on the structure, content, and conduct of a 
g i ven course?) 

f) the format used In reporting information (e.g., raw data? 
narrative ^summaries? statistical summaries? means? percent- 
ages? comparative norms? graphs? 

g) the reliability and validity of information (e.g., 
preliminary or concurrent research as part of the evaluation 
process? pilot studies? use of pre-tested instruments?) 

Up to this point we have drawn a kind of rough map of an 
approach to teaching evaluation vhich is based on the information needs 
of decision-making groups within a college or university. Along the way 
we have illustrated how different information needs might lead to the 
adoption of quite different evaluation procedures. In order to illustrate 
more concretely how this approach can be applied for improving instruction 
in colleges and universities, we would like to share two **case studies'* 
with you. Each of the *'case studies** is in effect a composite of several 
real-life situations, bringing together the best features of those rea) 
world (and therefore imperfect world) examples. 

A, The development of a student guide or handbook to 
courses and instructors. 
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B. Faculty use of alternatives to end-of -course eval- 
uation procedures for the improvement of courses and 
teaching methods. 

In each of these case studies, the emphasis is on selecting 
an evaluation procedure which will yield the kinds of information which 
the respective groups require i^ they are to make informed decisions which 
will improve the teaching and learning environment. Taken together, these 
**case studies'* illustrate how the improvement of coHeg*5 university teach- 
ing can be approached from two major vantage points; one hopes to the 
advantage of all. 
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S\'MPOSIl'M VI 



EMPLOYING CRITERION-KEFERENCED KEASUfcES 
FOR INSTRUCTIONAL IMPROVEMENT 



Chalrnian: Dr. Carmen J. Flnley 

Director of Assessment and 

Principal Research Sclent is t 
American Institutes of Research 



Participants: Dale I. Foreman 

Director, Test Development 
Vestlnghouse Learning Corporation 
I ova City, Iowa 

Jason Mlllman, Professor 
Educatlcnal Research Methodology 
Cornel 1 Unlve rsl ty 

Todd Rogers 

Assistant Professor of Education 
University of California, Riverside 
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3C0BE AN OBJECTIVE--REFERENCEE 
EVALUATION SYSTEM 



DeLIe I. Foreman 



{Dale I. Foreman is Director of Test Development 
at Westinghouse Learning Corporation » lova City, 
lowc. He received his B.S. degrees from Idaho 
State University and The University of Minnesota, 
aiid his Ph.D. degree from The University of 
Minneso.ta . ) 

Objective-referenced (or criterion-referenced) testing has 
become a key word in classroom and school evaluation. Many major testing 
groups have begun the development of test instruments that directly reference 
the test items to measurable objectives which are considered important out- 
comes of the learning process. School districts around the country are also 
beginning to focus attention on the implerr^ntation of a pilot program using 
some criterion-referenced instrument. Some areas have gone to the point of 
full implementation of a criterion -referenced program in one or more grades 
etnd subjects. 

Westinghouse Learning Corporation, in the last year, has join- 
ed the bandwagon. The test development staff has spent the last year devel- 
oping SCORE (School Curriculum Objective-Referenced Evaluation). SCORE is 
made up of four basic components, l) SCORE is objectives; 2) SCORE is 
objective-referenced items; 3) SCORE is custom test generation; and U) SCORE 
is scoring, reporting, and interpretation. 

The SCORE objective bfiuik was developed in such a way as to 
avoid reinventing the wheel. Objectives were obtained from many sources 
(lOX, Downers Grove, PLAN*, as well as individual school systems). These ob- 
jectives were screened to determine appropriateness and then the ideais 
incorporated into the SCORE objectives bank. In many cases, additional ob- 
jectives were written to fill existing gaps. All of the objectives used were 
organized into a three level hierarchy. The first level (tenninal objective) 
is a very broad goal-type statement which would not be appropriate for class- 
room use. The second level (intermediate objective) is more course oriented 
with a broad base. Finally, the lowest level (instructional level) is writ- 
ten for the classroom teacher. Each instructional objective covers a small 
segment of the content for a course* 

The SCORE item bank provides measures for all of the instruc- 
tional objectives in the system. These measures r&nge in formats from multi- 
ple-choice items to individually administered performance items. All items 
are reviewed in terms of face validity (Does the item measure the objec- 
tive?), appropriateness of reading level, emd relevance. 

SCORE works by involving the school district personnel in the 
selection of objectives that are considered critical for the children at each 
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grade level or in each grouping. This is done by choosing objectives from 
the cata]ogs and specifying the item formats desired fcr testing the objec- 
tive. Once the objectives are selected, a nachine-scorable booklet is 
created. This booklet contains items meas^aring only those objectives se- 
lected by the school district. 

If the school district wants to test the same with nK^re than 
one test to obtain grovrth measures this can be done also. Equivalent tests 
can be generated vhich measure the same objectives for use at different 
times during the year. The data can then be compared at the objective level 
to determine grovrth . 

This system has been designed to customize testing for any 
district. Some very small districts, however, can Join together in the se- 
lection of objectives to maintain economy. 

One of the critical parts of any system is the reporting 
scheme. The SCORE reports have been designed with several audiences in mind 
First is the teacher. The teacher is the one who must ^lan instruction, 
determine its effectiveness, and make critical changes which will better 
meet the needs of the students. This kind of information needs to be avail- 
able for the class and the individual students by objective. The SCORE cIps 
list gives a mastery report for each student by object. It also summarises 
each student's performance on all objectives and gives the teacher a summary 
of class performance by intermediate objective. The mastery of objectives 
is determined by a pre-specified criterion established by the school dis- 
trict, e.g. 80? mastery level. 

For the school principal or supervisor of instruction, a 
school building report is produced. This report gives information on the 
performance of each teacher's classes within the building as well as a 
summary for the school building. The summary data for each building is also 
compiled into a district report. This gives an overall report of performanc 
within the district and could be used as a school board report on instruc- 
tion and curricul\iin change. 

For the parents, there is a pairent letter. This letter pro- 
vides an ungraded verbal report of pupil performance. It states the inter- 
mediate objectives that have been mastered for the student aind those which 
have not been mastered. 

Other more standard reports have been included as a part of 
the total scheme. They include individual item analyses, reuok order distri- 
butions, and percentile distributions - 
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WILL TKE RIAL CRITERION- REFERENCED TEST 



PLEASE STAND UP? 



Jason Mlllznan 



Jason Mllljnan, on leave la^ Professor of Educa- 
tional Research Methodology at Cornell Univer- 
sity, is presently Cons\iltant-in-Residence, 
Instructional Objectives Escchange, Los Angeles. 



Several writers have expounded upon the distinctions between 
norm- referenced and criterion-referenced tests. Scores on the former are 
assessed by relating them to the test results achieved by some external 
reference groiqp (i.e., norm group); criterion-referenced tests derive their 
meaning by having scores; interpretable in terms of what the examinee knows or 
can do (i.e., performance). 

A review of the literature on criterion-referenced testi? re- 
veals that man:s' distinct concepts of such tests are being employed. The 
many types of criterion-referenced tests can be divided according to the 
purpose which they are 'ntended to serve. Descriptive interpretations and 
decision-making are two major, distinct uses to which criterion-referenced 
information is put. Exaaqples of comaon applications of criterion-referenced 
test uses and their relationship to the descriptive and decision-making dis- 
tinction will be provided. Content standard, universe -de fined, domain- 
referenced, ob^lectives -based, and mastery tests will be discriminated. 

Of utmost importance is the fact that appropriate item selec- 
tion criteria, test construction principles, and validation techniques de- 
pend ultimately on the nature of the inference desired by the educator. 
Attention will be given to the different ways such tests mi^t best be con- 
structed and validated for both descriptive and decision-making uses. 
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CRITERION-REFERENCED MEASURES FOR INSTRUCTIONAL 



IMPROVEMENT: ARE THEY EMPLOYABLE? 



W. Todd Rogers 

(W. Todd Rogers is Assistant Professor of Educa- 
tion at the University of California, Riverside. 
He received his 5. Sc. and M.A. degrees from the 
University of British Columbia and his Doctorate 
degree from the University of Colorado.) 

Instructional improvement depends upon many factors, the 
first being the generation of relevant information which can be used to 
describe the efficacy of the present instructional program or programs. 
Second, the curriculum supervisor, principal, department head and/or 
teacher must be willing to change if the information reveals an inadequate 
instructional program. Thirds, the responsible parties must be able to 
analyze alternatives and be able to implement suitable alternatives. 
Fourth, all must be willing to systematically repeat the cycle. The topic 
of the present symposium pertains to the generation of relevant information. 
Such generation is no guarantee that instructional improvement will neces- 
sarily take place. 

The intention to employ criterion-referenced measures for 
instructional improvement is underscored by the growing demands for in- 
formation on how well schools are fulfilling their responsibility to edu- 
cate children. The difficulties associated with developing such measures 
in learning areas other than the minimum, basal skills and with their 
psychometric properties notwithstanding, one cannot fail to appreciate 
their importance. It is doubtful that our ignorance can be eradicated by 
a testing program yielding normative information. Available standardized 
tests, with their national focus, sample only that portion of the school 
curriculum common to all geographic and political regions of the country; 
they are unlikely to encompass the scope or penetrate to the depth of a 
particular instructional program. As they are now constituted, these tests 
provide normative information about pupils which can be used for classi- 
fication and selectivity; they do not provide adequate knowledge about 
what skills and knowledges have been acquired by a pupil. 

Criterion-referenced tests are deliberately constructed to 
yield measures that are directly interpretable in terms of predetermined 
performance standards. Such measures might be used to determine th^ effi- 
cacy off a particular instructional program in meeting a set of local in- 
structional objectives or tc determine whether a pupil has mastered skills 
considered prerequisite to his commencing a new learning task. The logical 
appeal of criterion-referenced measures for instructional improvement 
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resides in the notion that such measures should serve the function of moni- 
toring instructional programs. By providing relevant, more readily inter- 
pretable feedback to both teachers and students of the intent as well as 
the degree of success in achieving the objectives or goals of the program, 
clearer information will be available as to what has and has not been 
learned. For example, the meaningfulness of a particular group criterion- 
referenced score (e.g. 65% of the grade 4 pupils know the basic multipli- 
cation facts) is dependent upon an agreed standard (e.g. 100% mastery of 
the multiplication facts at the fourth grade level) and not comparison with 
other groups. Similarly, the meaningfulness of a particular individual's 
score is dependent upon comparison with an agreed, predetermined level of 
mastery on a task prerequisite to further learning and not upon comparison 
with other testees. 

Methodological Concerns 

To set in advance what constitutes an acceptable set of 
goals or objectives or what constitutes an acceptable level of performance 
is more complex than one might initially perceive. There are few com- 
petencies for which complete mastery is possible. For the remaining, 
several questions might be raised: What is mastery? How does one justify 
a criterion other than 1007o (say 80%,) as a cutting score or indication of 
minimal competence? What are the procedures or methods for arriving at 
such standards? How does one construct the items or exercises to obtain 
the necesf^ary measures? How do changes in wording of an exercise influence 
the examinees performance? Given these concerns, are there procedures now 
in use that have gained some respect? 

Criterion-referenced measures have been successfully con- 
structed for certain basic skills (e.g. the multiplication facts) and in 
areas where psychomotor objectives (e.g. driver training) are involved. 
Minimal levels of satisfactory performance are established on empirical 
grounds or logical grounds which are easily agreed upon. The General 
Educational Development (GED) tests, which use the actual achievement of 
high school graduates to establish minimum standards, is an example of an 
empirical approach to setting standards. Sources of empirically based 
criteria that may be appropriate for a local instructional program include 
the reported results of national and state assessments. The use of expert 
judgments is another common practice for setting criteria- -although the 
questions "who are the experts?" and "how are these judgments combined?" 
are not yet fully answered. Task analysis of the skill or understanding 
to be learned might be useful in establishing minimum standards for each of 
the component subtasks. Examples of this approach have been given in 
arithmetic; however, task analysis appears much more formidable in other 
subject areas as well as in the higher learnings of algebra. In defining 
the subtasks, are all identified? Will the emphasis be placed on the prod- 
uct of each subtask, rather than on the processes themselves? 

There are available several resources for obtaining ob- 
jectives and objectives-referenced exercises that have proved acceptable 
in several situations. Although this does not eradicate the prespecified 
criteria setting problem, it does provide valuable information which, on 
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the basis of reaction to the results obtained may have post hoc criteria. 
Perhaps this might be a necessary first step in implementing a full cri- 
terion system. 

A second major concern associated with the use of criterion- 
referenced measures revolves around their psychometric properties. -The 
present paper will not address the points commonly considered except for 
questions of content validity. Several references on the psychometric 
properties of these tests are included as an addendum for those people 
unfamiliar with this aspect of criterion measure and interested in pursuing 
the matter. 

Can instruction be improved through the use of criterion 
measures? Until a sufficient number of successful examples at the various 
levels (national, state, local) of education have been developed and 
replicated, caution should be employed. Before the potential contribution 
of criterion measures is fully realized, much developmental work must be 
done. Furthermore, in-service educ;»t:ion will be needed to help insure 
good use is made of the process and available products. 
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Chairman: 
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Associate Dean 
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Association of California School Administrators 
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EVALUATION CF ADMINISTRATIVE PERSONNEL 



Wiarren Linvi I le 

(Wcrren Linville is Superintendent of Schools 
for the Montebello Unified School District, 
Montebello, California. He received his B.S., 
M^A. and Doctorate degrees from the University 
of Southern California. He has tuuyht grodes 1 
through the graduate level and served as on 
Administrator, grades K*14 in the public schools.) 

Most of us can remember when the evaluation of a school od- 
ministrator depended on such things as having a winning football team, membership in 
a service club or being on all-round nice guy. Sometimes it depended on the things 
he didn't do, like getting too involved in day-to-day classroom instruction, questioning 
vested interests or rocking the ship with new ideas. Of course, there were written 
evaluation instruments- usually in the form of check lists covering critical items such a> 
"dresses appropriately", "supports and participates in professional organizot ions" and 
"displays enthusiasm'', in the long run, however, the administrator who ployed hii cards 
close to his vest and minded his "do's" and "dont*s" was pretty much assured of a long 
tenure. 

With the advent of the Stull Bill the opportunity for operational 
profess^^onal ism has mode an appearance. Accountability is the word of the day. It has 
goined public support because we all like to feel that everyone else is accountable for 
everything that goes on in the world around us. The day of administration being a self- 
fed, self- perpetuating enfi^y is about over. Instead we are beginning to reolize that the 
primary justification for Administration is to promote ond support the cttolnment of the 
educational goals In the community we serve. 

Obviously a more systematic approach involving a wider audience 
must be involved. A checic list assumes that the list covers the brood areas of competence 
but leaves out perhaps the most Important and far-reaching element of professior.ol evalu- 
ation kn0wn as self-renewal. The check list may point out good, not vo good, poor or 
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very poor os viewed by t^e sc^p^rv isi^^g evoluo^or bo* does l5*He ♦o v«rve the e»"jlL»otee in 
the oil i'^pc^o'^t area of self- re^e^ot . A systemo*Ic opprv-^sc^ deslg'^ed through brooder 
on thp i'^r^ed'C*e supe''»!SO' o'^d ir-^cKt'^a a ^ide^ OLd'e^^ce ^aiies it possible ^or 
the e*oluotee to gain Irsigntj leoding t© real and rr.eoningful self-renewol . 

Recognizing that ♦he dimensions educo^ionTl lecderihip ore 
much brooder ond deeper than viewed, *he Montebello Unified Schoct District launched 
o S|S*ematlc approoch ♦c ad'^inistrator evoluotion in the spring of 1971 . 

The plon called o Monogement Sup>port Syster^i contains severol 

themselves, '2) following an ,essmer>t cf individual needs, the odm inistrator has the 
primary responsibility evol'./oting the findings, ond '2) the od*^' i n i st ra tor ond his 
Supervisor jointly og»ee upor on 1 nd I » 1 duo I i red inservice plan for jmprovement. 

The s^ste*^, still in the process of development ond refinement, 
bego*^ ^ith ogreemer* irpor brood gools ♦hat v^ould cover all conceivoble activities. 
These goals ^ere eventually cost in*o five Admi n Ivtrat 1 ve Role Statemenf* ogreed upon 
br all odm i n is* rotors «n the dis^Mct, The Role Statements cover the areas of ^ ) evolu- 
o*ing and mprovement of v»udent learning^ (2) personal picms for p^^ofession»at crov^th end 
sel f- renev^al , '3) staff ond student commitment to the school's educotionol purposes; 
^4/ commitment o^her school personr>el, the community end profession at large; ond 
'5 effective manogement of resources human, material, time and space. 

Although consensus upon the role statements was on importont 
first step, the stote'^^ents were of little use until they could be translated into operational 
terms. Each of the t^enty-se^en principoU o^ the district along with district odminis- 
trators, selected the specific role ^^otemcn^ ♦hey wished to develop. For more than a 
yec. ♦^'^cmi ieviewed the literoture of private ond public odmlnijtrat'cy^, o'^alyzing re-^ 
seorch Instrurients ond synthesizing processes for assessing competency In each of the 
Role Statements. A unique requirement of the process was thot asscssrr>ent instruments 
hod to ^ designed to somple the populotlon to be served by the odministrotoi^ ^or 
exomple, the assessment of the le^el of portl c ipoti ve manogement ot a local site is mode 
by the school staff served, not by on odmlnistrator at the district office. 

Following volidotic>n of the selected operational procedures, on 
assessment is mode by eoch odministratc>f . The results of this ossessment are onalyzed and 
evoluoted by the odministratof who then develops o proposed plon for perscx^ol inservlce 
activities. This plon is mu^uolly ogreed upon by the odrr^iniit rotor ond his immediote 
supervisor, resultir>g in on evoluotion system bosed on the Individuol profile of profes- 
sional needs ond inservice octlvities designed for ^^le odmlnistroto* in his specific 
ossignment . 
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THE ROLE OF THK PRIhiCihAL IN THfc AREA OF EVALUATION 



AS AN EVALUATEE 



Dorothy L. Tillinghast 



Dorothy Levesoue Tillinghast is Principal of 
Lcmay Street School, Van Nuys, Los Angeles City 
Unified School District. She received her B.A. 
degree from the University of California, Los 
Angeles, her M.A. degree from California State 
University, Los Angeles, and her Ed.D. from the 
University of Southern California. 



In California, vrith the Stull Act a very real part of our 
lives, we face mandatory evaluation. 

As administrators, we have continual work in evaluation in 
terms of being an evaluator. This paper will attempt to look at the 
administrator's role as an evaluatee. 

My frame of reference dealing with the Stull Act comes 
from my experience in the Los Angeles City Unified Schools, so this is 
what will be used for this paper. For example, the Personnel Assess*- 
ment Department of the Educational Goals Section suggests that in Los 
Angeles eight are&s of an assessment of performance have been selected 
that have reference to Board Rule 4310 (Stull Act as pertaining to the 
evaluation and assessment of management and service personnel). They 
are: 

Administration Instruction 
Business Management Personnel Management 

Community/Public Relations Plant Management 

Human Relations Ripil Services 

In Los Angeles it is the evaluatee who determines what 
areas he will work in and what the assessment technioues to be used will 
be. We have been encouraged to make it practical - to set priorities - 
and to make what we select narrow enough to be feasible, to be something 
that can and will be accomplished. By way of illustration, we will 
follow just one area of possible performance. Let's follow through what 
might be done in the area of Cortttnunity/ Public Relations. 
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1. ASSESSMENT OF STANDARDS OF FKRFORMANCE 




Plans for Meeting Standards, Goals, Objectives 
OX uisuncu^ A rest ^ ocnoox^ or uiixce 


Assessment Technioues 

uO De USCQ 


COMMUNITY/PUBLIC RPXATIONS: Organize an 
effective system of providing continuous 
newsworthy items for local newspapers. 


Assessing of minutes, 
schedules, and records. 


2. ASSF.SSM?,NT OF CERTIFICATKD PF:RS0NNF,L C0MP?.™CE 


Plans for Performance of Duties as Related to 
Class Description 


Evaluation Procedures to 
be Used 



a. Identify ten opinion leaders within 
school service area. 



Hold discussions with leaders to determine 
interests and attitudes of community. 

Review with administrative staff and 
journalism advisers ways to meet the 
needs of, as well as to identify the 
attitudes held by, the community. 

Cooixiinate planning sessions for the 
purpose of: 

(1) Determining policy of newspapers 
and deadlines 

(2) Developing a reporting system from 
each department 

(3) Preparing and submitting of reports 

(4) Pictorial support 

Review plans with staff and students to 
obtain reactions and recommendations for 
improvement of plans. 

Invite local press to school for a public 
relations luncheon and explain school plan. 



g» Implement plan^ 



h. f]valuate program. 



Assessing the list 
and discussing how 
list was compiled. 

Checking minutes of 
meetings. 

Reviewing schedules 
of meetings or 
minutes. 



d. Reviewing progress 
made following the 
sessions. 



a. 



b. 



c. 



e. Assessing reactions 
and recommendations 
of staff. 

f . Assessing interest 
and attitudes toward 
planning . 

g» Reviewing progress 
made and assessing 
interest and support. 

h. Observing. 
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We need to utilize all possible avenues of learning how 
to evaluate others. There is work bein.^ done in the areas of observing, 
conferencing, and evaluating for effective supervision. Dr. Madeline 
Hunter at U.C.L.A. and Dr. John H. Hansen at Florida State University 
are two who have seme interesting ideas in these areas. 

By becoming adept in evaluating others, we learn to 
evaluate self and grow from it, thereby practicing self -renewal. 
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rFt'BLF:Ms IN Tin: rvALrAiioN of administrative personnel 



Edward V. Beaubier 

Every successful manager-leader, whether in business or 
education* finds it advantageous to periodically check his intuitive 
insigJit into the operailon for which he is responsible. This assessment 
raay range from physical facilities to aesthetics, morale, and/or his 
leadership-managerial style. Usually, or far too frequently, the method 
of assessnvent is intormal, consequently more subjective than objective for 
the manager has a tendency to observe or perceive only what he wants to 
observe . 

The informal or over subjective type of evaluation of 
administrative personnel presents a major problem. This problem seems to 
come about when the following appear: 

1 . There is a lack of community substantiated school 
district goals and objectives. 

2. School district has not set definitive administrative 
job descriptions. 

3. No mutually agreed upon exr»ectat ions for members of the 
admirvistrat ive team exist — accountability. 

4. There is a lack of pre-determined evaluation criteria. 

5. There is a lack of valid data for evaluation of 
the administrator . 

More and more modern manager-leaders are finding it to 
their personal advantage, as well as leading to the better achievement of 
the institution's goals, to utilize somewhat more formalized and objective 
assessment techniques to validate their perceptions or to provide them 
nev insights into areas which are problems or could soon become problems. 

However, one major problem area abouL which much has been 
written is the type of objectives or goals set and the way in which they 
are developed. Odionne, in a recent American Association of School 
Administrators publication, identified twenty of the most common errors 
in goal setting. A few of these errors follow: 

1. The manager doesn't clarify corrmon objectives for the 

whole unit. 
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2. He doesn't use prior results as a basis for using 
' to find new and unusual conb ina t ions . 

3. lie doesn't clearly shape his unit's common objectives 
to fit those of the ^ ivjk T unit of which he is a part. 

4. ^llc'^^ tvs or nrr^ 'Iri^livi duals to believe thcnsclve^; 
n.-; '.,^*.' \- for doing exactly the same thing when he knows that having one 
responsible party is better. 

5. He tacitly implies that flcaainp hir is what really 
counts, rather than achieving the job objective. 

6. Ke doesn't probe to discover what his subordinate's 
:r.^:rjir: for goal achievement will be. He accepts every goal without 
criticism and without seeing a plan for successful achievement, 

7. He ignores the very real obstacles the subordinate will 
face in achieving his goals, including many emergency or routine duties 
that consume time. 

8. He fails to set intcmediate target dates (milestones^ 
by which to measure progress of subordinates. 

9. He is rigid in forbidding the abandovuiient of goals that 
prove unfeasible or irrelevant. 

10. He doesn't reinforce successful behavior or investigate 
unsuccessful behavior when goals are achieved or missed. 

From the responses on an Educational Research Service 
questionnaire, "For what purposes do you evaluate administrative and 
supervisory personnel?'* administrators indicated evaluations have been 
applied as follows: (Parentheses indicate number of systems reporting) 

Identify areas needing improvement (77); Assess present 
performance in accordance with prescribed standards (70); Establish evi- 
dence for dismissal (60); Help evaluatee establish relevant performance 
goals (60); Have records to determine qualifications for promotion (55); 
Determine qualifications for permanent status (35) ; Determine qualifications 
for regular salary increments (9); Comply with board policy (8); Determine 
qualifications for merit pay (3); Comply with State law/regulat ion (3); 
and Point out strengths (2). 

Experiences in the Association of California School Admin- 
istrators Project Leadership indicate that problems in evaluation of admin- 
istrative personnel can be reduced when an individual administrator (1) has 
mutually agreed upon product personal and process goals related to his job 
description and (2) has carefully designed a monitoring program developed 
cooperatively with those involved. We like to think of evaluation as being 
used for improvement — not to prove. 
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SYMPOSIUM VIII 



EARLY CHILDHOOD: FABRIC, FANTASY AND FACT 



Chairman: Dr. Mabel C. Purl 

Director of Research and Evaluation 
Riverside Unified School District 



Participants: Robert C. Calfee 

Professor of Education 
Stanford University 
and 

Kathryn Ann Hoover 
Research Assistant 
Stanford University 

Frank Sata, Architect 

Member, Early Childhood Task Force 

Instructor, Pacific Oak College 

Frank Delavan 

Manager, Early Childhood Education 

Management Tean 
State Department of Education 
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POLICY A\D PR.\CTICF: IN EARLY EDUCATION RESEARCH 



Robert C. Calfee and Kathryn A. Hoover 

(Robert C. Calfee is Professor of 
Education at Stanford University, 
Stanford, California. He received his 
B. A. from the University of California, 
Los Angeles* and both his M.S. and 
Doctorate degrees from the University 
of California at Los Angeles. Kathryn 
Hoover received her B.A. and M,S. degrees 
from California State University at 
Long Beach and is presently working on 
her doctorate at Stanford University.) 

A considerable body of research now exists concerning the 
effects of formal and informal education for children between the ages of 
three and nine. What is the value of this literature for those respon-- 
sible for policy decisions? How useful is it for practical purposes in 
t.^e implementation of early education programs? What form should future 
research take in order to be most helpful to planners and decision makers? 
The purpose of the present paper is to discuss these questions in a general 
way, and illustrate the key points with pertinent examples. 

Planning an educational program requires that we give 
thought to existing needs and resources, and to future goals. Since needs 
seem often to outstrip resources, we must establish priorities, look for 
ways of using available resources as efficiently as possible, and seek to 
develop new resources. The immediacy of present needs can easily over'^ha- 
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dow and forestall any serious consideration of future possibilities; for 
better or worse, educational goals tend to focus on what is right now rather 
than what might be- 

An objective, analytic, rational and open-minded search for 
facts - which typifies research at its best - can help in characterizing 
and_ describing needs and resources, in evaluating the relative efficiency 
of different programs for resource allocation, and in establishing^ the 
fe a sibility and likely outcomes of innovative programs. To do all this, we 
need research designs that incorporate adequate control over relevant vari- 
ables, that can be reasonably generalized to situations of direct conce rn , 
that are sufficiently comprehensive in scope to encompass a fair range of 
alternatives, and that are multivariate in character . 

Unfortunately, a great deal of the available research is 
inadequate by one or more of these criteria. Much of it follows the 
pattern of Approach A versus Approach 3, measured by change from pre- 
to post-test on a single criterion variable. Acutal control is usually 
poor; and it is difficult to determine what the actual "treatments" 
were. Generalizations are drawn from populations of unknown or inappro- 
priate character; the reaction of orphaned infants to institutional 
treatment is of questionable relevance in deciding whether four-year 
olds should be enrolled in public schools, for instance, though such 
comparisons have been suggested. 

These are not new complaints; in part, they represent 
shortcomings on the part of researchers, and in part they result from 
unreasonable policies of public funding agencies. Research and evalua- 
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tion efforts generally receive low priority, little authority and great 
expectations. 

Nevertheless, there are reasons for optimism. First, 
researchers, evaluators and funding agencies are increasingly cognizant 
of the shorlcomings of many past efforts, and aware of the critical 
elements in some successes. Second , the techniques needed for more 
adequate planning, design and analysis of educational research are becoming 
increasingly available to the profession as a whole. Thirdly, we can find 
several instances in which research has led to reasonably clearcut infor- 
mation of use to policy makers and program administrators. 

Three case studies pertinent to early education illustrate 
the points made above: (a) What is the appropriate age for a child to 
enter school? (b) What is the effect of early intervention for dis- 
advantaged children? (c) Does it matter what instructional program 
you chos2 (for reading, preschool education, etc.). 

These case studies will be used to point up the practical 
consequences of inadequate research methods, to demonstrate the role of 
analytic planning in clarifying the question, and to suggest that a few 
questions can be reasonably well answered at the present time from past 
research. 
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THE "WHOLE CHILD": A TASK FORCE FANTASY? 



Frank T . Sata 

(Frank T. Sata, Bachelor of Architecture, University of 
Southern California; member Early Childhood Education 
Task Force; testified before General Subcommittee on Educa- 
tion; Staff Architect Early Childhood Education Study of 
Educational Development Center of INewton, Massachusetts; 
Teacher at Pacific Oaks College; Resource Consultant in 
Head Start and related community action programs through- 
out the country. Private architectural practice. 

Wot-ta weekend! Little did I realize when I agreed to Mabel Purl to 
appear as a panel participant that I would re-experience cram-pains, finger cramps, and 
"gotta finish that paper" blxjes. Architects have had notorious reputation for being poor 
spellers and grotesque interpreters of the I'.nglish language. These are by my own admis- 
sion the only virtues endowed upon me by training and there ends the similarity between 
what I am and the professional license i hold. 

On Saturday morning, Septen^ber 22, 1973, 2:00 a.m., I decided to con- 
struct the puzzle from my many notes of these past months. The boys and I had watched 
G part of TORA, TORA, TO PA this evening and I was wondering what passed their minds 
OS they watched the bombing of Pearl Harbor. It appeared that in a very short time, one 
generation to be exact, a moment In history fhat had direct influence on their parents 
might be nothing more than novel entertainment. There was concern, however, of the 
repeated interruption by the commercials. And so with the Coca Cola song in the back- 
ground, I continued to construct the puzzle. 

September 5, 1973, 5:00 p.m. I ventured to my studio, hoping to be in- 
spired by the quiet of my studio to complete this "brief." An associate had the radio 
playing and to my dismay, I listened to the Coca Cola song in total. This is not an 
attempt to equate Coca Cola and education but it's a helluva testimony of the medias 
that supposedly embelishes the human environment today . The merits of Coca Cola might 
be far better oppreciated than the merits of education, A bit presumptuous on our part 
to believe we have the answers necessary to program the child to cope with all of tomor- 
row's problems. To pass on data from generation to generation has always been an 
aspiration worthy of great accolade by the intellectual mind, past and present. What is 
data and therefore relative to its own time-motion period versus what is knowledge and 
worth passing to the next generation are notions requiring proper evaluation. 
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To look at the "whole child", let us discuss the "whole being". My own 
simplification of the whole being might look like a simple balance or teeter -totter • You 
can place the brain on one side, the body on the other side and it is in perfect balance. 
Remove a sensory component from the brain and the body compensates. Remove a portion 
of the body and the brain compensates. In other words, the wholeness I refer to is the 
strength to compensate and balance. It is this strength that provides the power to (earn 
and grow, seek and discover. So within the concept of balance we have absolute and 
abstract, known and unknown. Within the v/hole being we must allow for the existence 
of abstract components or we are truly "robots" with varying faces. An example of an 
abstraction is time. Time has a meaning which belongs to each individual. It is the 
quality of life we refer to, not the quantity of years. 

The educational task is to take the "whole child", provide the knowledge 
and allow the child to grow as a whole being. The delicate balances are under constant 
stress by society*s prejudices and conflictions. Education as an institution must recognize 
its own bureaucracy, its elitism, its protective isolationism. Education through the media* 
is captive to the greed of capitalism. Education in the home is practically non-existent. 
Education through the physical environment is sad to say the least. If one carefully looks 
around, it is apparent who receives first priority in the cities; the car! Technology is 
a force that must be dealt with. We have created this force and we stand m awe of it. 
We consume more power because we are convinced we need it. We pollute our air, pave 
the earth, run, run, run, run, 

Frank Lloyd Wright referred to students being educated but not cultured at 
all. The educational process embodying cultural consciousness will sound in harmonic 
unison the advent of the whole man, , A whole man, in balance, who can trust with 
meaning and love with feeling. Tomorrow without learning is another day. Tomorrow . 
with meaning requires a child today, a "whole child". The child is real, not a fantasy, 
let us give the child a chance to grow. 
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EARJY CHILDHOOD EDUCATION 



Frank E. Delavan 



(Frank E. Delavan is the Manager of the Early 
Childhood Education Management Team for the 
California State Department of Education. He 
received his B.A. and M.A. from Sacramento 
State Collsge and his Doctorate from the 
University of California at Berkeley.) 

One of Dr. Wilson Riles* first acts upon assuming the post 
of State Superintendent of Public Instruction in 1971 was to appoint a 
special Task Force on Early Childhood Education. This 24-member task force 
was to make sweeping recommendations for revitalizing public educational 
programs provided during the initial years of a child's school experience. 
Such a revitalization v/as desired (a) to promote the development of the 
basic skills essential to further success in school and life, and (b) to 
begin the process of restructuring all of public education, K-12. 

The Report of the Task Force on Early Childhood Education 
envisioned programs encompassing one year prior to kindergarten through 
the third grade level, or its equivalent. Its recommendations were 
incorporated into a proposal adopted by the State Board of Education and 
then drafted into a bill (S.B. 1302) for consideration by the 1972 
Legislature . 

Late in the legislative process, however, the original bill 
was amended so as to defer the pre-kindergarten year feature of the pro- 
posal for reconsideration by the Legislature in 1975. Another modifica- 
tion of the bill related the ECL program fiscally with the Miller-Unruh 
Reading Program. The amended bill was passed by the Legislature, signed 
by the Governor on November 27, 1972, and became effective on March 5, 1973. 
Th^B bill appropriated $25 million for ECE programs during 1973-74, 
^nO million for ECE programs during 1974-75, and $250,000 for State Depart- 
ment of Education support operations during the balance of 1972-73 and 
all of 1973-74. In order to provide assistance to ECE schools in planning 
thfeir programs and to provide for a system and criteria for the review 
and approval of the school-level plans, an Early Childhood Education 
Management Team was created in March-Apr'il* 
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Although incorporated into the new statewide consolidated 
application procedures regarding most federal and state categorical alloca- 
tions to school districts, S,B. 1302 also requires that a comprehensive 
school-level plan be submitted for approval by each participating school. 
These school-level plans for ECE programs are to be concerned with 
individualized instructional programs using diagnostic-prescriptive 
approaches which are based on identified needs. The bill requires 
that parents participate in the planning, implementation, and evaluation 
of the programs. The programs are to reduce the adult -pupil ratios to 
1:10 in the participating schools and to provide staff development and 
parent education activities necessary to help teachers and parents fulfill 
their roles in the program. These programs also are to stress, as needed, 
diagnostic, referral, and follow-up efforts in the area of health. 

By September 13, 1973, the Early Childhood Education 
Management Team had received 1,019 school-level plans for ECE programs. 
Eight additional schools were eligible to submit such plans, but elected 
not to do so. Two schools submitted plans which did not meet the require- 
ments of the program and decided not to continue developing their plans 
accordingly. 397 school plans were reviewed and approved as submitted. 
The remaining 620 school plans were reviewed and requests were made for 
addenda or revisions. All but 30 of these school plans had been revised 
accordingly and approved for a total of 987 approved ECE programs on that 
date. Since September 13, most of the pending school plans have been 
resolved so that approximately 1,000 schcols will be initiating ECE 
programs for 1973-7^+. 

Admittedly, there are some schools entering this program 
which are experiencing anxieties and problems.. However, in general, the 
ECE programs in the participating schools are generating considerable 
enthusiasm on the part of parents and teachers, and S.B. 1302 is proving 
to be the most' exciting thing to happen to kindergarten-primary education 
in California in over a generation. 




88 



SYMPOSIUM IX 



STAFF DEVELOPMENT PROGRAMS THAT MAKE A DIFFERENCE 



Chairman: Robert Hocker 

Director of Pupil Personnel 
Alvord Unified School District 



Participants: Thomas Gordon, President 

Effectiveness Training Associates 
Pasadena 

James L, Olivero 
Executive Director 

Nueva Day School and Learning Center 
Hillsborough 
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riTFAll.S AM> FKATFAl LS IN STVLL ACT I MPLFMINTAT I OK 



Dr. Richard J. Shavelson 
Actlnp Assistant Professor 

of Educat ion 
School of Educat ion 
Stanford University 



Part i c i pant s : 



David C. Berliner 
Director of Evaluation 
Far Wes: Laboratory 



Eramett R. Berry, Jr., Superintendent 
Riverside Unified School District 
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TIfK STULL ACT IS ONLY A TOOL 



Rav Berry 



(Ray Berry is Superintendent of Schools in Riverside 
Unifie<J School District, Riverside, California. He 
has been Superintendent since 1968, Prior to that 
time, he worked in the district in helping initiate 
development of personnel and instructional functions 
appropriate to a decen. ral' zed, integrated school 
system.) 



My contribution to the Stull Act panel stems from experience 
with evaluation programs at district level and not from formal research 
on the Act implementation. Much of my statement is, therefore, more 
a matter of personal opinion than of fact. 

Nevertheless, experience can be a hard teacher, especially 
when dealing with the difficult business of evaluation of one human 
being by another. 

I certainly believe that evaluation is badly needed and neces- 
sary. Moreover, I am convinced that useful evaluation between people 
can occur. 



I do not accept that a paper product of guidelines, mandates, 
objectives, and procedures which are imposed upon an organization such 
as a school district, from either outside or inside, will do the job. There 
is no more of a tried and true method of evaluation than there is of teach- 
ing reading- -and for similar reasons. Human beings are just too complex 
and contrary ! 

If months and even years of preparatory foundation work does 
not occur, then any system of evaluation policies and regulations is not 
likely to really succeed. In fact, such a system may prove counter- 
productive. 

By foundation work, I mean that the most of the following com- 
ponents must be an integral part of the organization- -in this case, the 
school district: 



1. The organization itself cannot bv^ purely hierarchical in struc- 
ture. Power, authority, and responsibility which are focused 
entirely at the top and imposed downward provoke reaction 
which suppresses, or even defeats, true professional eval- 
uation. 
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2. Everyone must have a very clear understanding of his role and 
should ha\-e had a hand in arriving at that role description with 
his or her immediate supervisor. 

3, The purpose of evaluation is to improve service. Most of '!:he 
time it can and should be a positive rewarding experience for 
everyone concerned, 

4. An alternate to dis^missal for less than adequate performance 
is desirable. One such possibility is retention or even re- 
gression on salary schedule steps, 

5, Measures of performance are only ONE part of an evaluation 
process. The individuals who are being measured should have 
been significantly involved in making decisions about the ex- 
pected performances and their measures, 

6, Sometimes evaluation needs to occur other than by schedule. 
Education is a people process. People do not fall into neat 
niches. Things happen to people. Usually, certain signals 
occur which indicate deviations or stress. Procedures need 
to be available for responding to such signals promptly, ef- 
fectively, and constructively, 

7, We have done a fine job of protecting rights and privileges of 
employees and other adults. We haven't done too well for the 
kids. By "we, " I mean the courts, the laws, the school dis- 
trict authorities, and the employee organizations. Occasion- 
ally, we are going to have to agr ee that an employee is wrong 
and doesn't belong near children. Of course, this is what the 
argument is all about, but I contend that we haven't really 
matured yet to acceptance of such responsibility jointly, 

8, Middle management personnel are keys to successful evalu- 
ation, li their skills and attitudes are inappropriate to the 
process, evaluation cannot succeed. Therefore, training 

and evaluation of these key people must be first and foremost! 

9, Reasonable decentralization of the organization hierarchy with 
concomitant allocation of responsibility and some self-deter- 
mination helps set the atmosphere for professional, objective 
evaluation procedures. Such organizational efforts require 
long, determined involvement and commitment on the part of 
everyone. The task is administratively more difficult, but 
performance potential seems to be extended significantly. 

The Stull Act is a necessary authoritative commitment to the 
process of evaluation, Hov^ever, it is only a door-opener. The real 
responsibility for making a people -p rocess work effective between people 
belongs at the people level. These are the school board, the superin- 
tendent, and especially the in- school staffs and the students and citizens. 
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The words of the law or policies or behavioral oojectivcs and 
tho like arc nn^roly to<»ls. Dr\el<^pmcnt and use of such tools, wi^^->out 
sonsit'vity or mutual respect and appreciation or honest commitnient, 
are a cop-out. Such failure is considerably beneath the level of profes- 
sional dignity that citizens should expect from those who work witli 
chi Id ren. 
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